As the amount of data in computing systems continues to increase, there is a strong desire for improvements that allows the datasets to be efficiently processed. DRAM (Dynamic Random Access Memory) and the like are often too small to efficiently process large data sets. Algorithms that process data out-of-core (using Hard Disk Drives (HDDs)) tend to be slow.
One potential solution is to introduce flash memory into the computing systems. Flash memory is faster than HDDs and has the capacity to accelerate dataset analysis. Even though flash memory can improve the processing capability of computing systems, flash memory has several problems that impact performance.
For example, conventional data structures are designed assuming that random changes or random edits can be performed quickly and without penalty. Flash, memory, however, has a penalty associated with small edits. Small edits in a flash memory require the edited page to be copied forward to a new page. The previous page must be eventually erased before it can be reused. More specifically, data in a used area or page of a flash memory cannot be simply overwritten in a conventional flash memory. Rather, it is necessary to erase the page before writing the data. This is the reason that small edits to a page in the flash memory are simply written as a new page.
This process causes both a performance penalty and a lifespan penalty. This process results in multiple reads and writes (thus the performance penalty). The lifespan penalty occurs because flash memory can only be written or erased a limited number of times before wearing out. Further, flash memory is typically erased in large units.
This creates additional problems when implementing data structures in the flash memory. Every time a change is made to data that is stored in the data structure, there is a potential for multiple writes and erasures. Systems and methods are needed to improve the performance of flash memory and to improve the performance of flash memory and to improve the lifespan of the flash memory.